Learning Light-Weight Translation Models from Deep Transformer

Recently, deep models have shown tremendous improvements in neural machine translation (NMT). However, systems of this kind are computationally expensive and memory intensive. In paper, we take a natural step towards learning strong but light-weight NMT systems. We proposed novel group-permutation based knowledge distillation approach to compressing the Transformer model into shallow model. The experimental results on several benchmarks validate effectiveness our method. Our compressed is 8 times shallower than model, with almost no loss BLEU. To further enhance teacher present Skipping Sub-Layer method randomly omit sub-layers introduce perturbation training, which achieves BLEU score 30.63 English-German newstest2014. code publicly available at https://github.com/libeineu/GPKD.